A message queue is a form of communication used in distributed systems and software architectures where different components or processes need to communicate asynchronously. It is a mechanism that allows messages (packets of data) to be sent and received between different parts of a system without the components being directly connected or synchronized in time.
The basic idea is that a sender (producer) can place messages in a queue, and a receiver (consumer) can retrieve and process those messages from the queue. This decouples the sender and receiver, enabling them to operate independently and at different rates.
Some key concepts associated with message queues include:
Message queues are used in various scenarios, including:
In Linux, the kernel itself provides a set of message queue facilities that are commonly used for inter-process communication. These facilities are part of the System V IPC (Inter-Process Communication) mechanisms, and among them is the Message Queue.
It's important to note that System V IPC, including Message Queues, has been part of Unix-like operating systems for a long time, but it's not the only inter-process communication mechanism available on Linux.
Many newer Linux systems often provide an alternative IPC mechanism called POSIX Message Queue. POSIX Message Queues are generally considered more modern and flexible than the older System V Message Queues, but both are available on many Linux systems.
Popular message queue systems include Apache Kafka, RabbitMQ, ActiveMQ, and Microsoft Message Queue (MSMQ). Many of these systems offer additional features such as durability, scalability, and support for different messaging patterns, making them suitable for a wide range of distributed systems.
In microservices, message queues foster independence and enable asynchronous communication. They act as buffers during service failures, ensuring message delivery and enhancing system resilience. Message queues support event-driven architectures, maintaining message order and facilitating microservices orchestration. They contribute to scalability and responsiveness, allowing microservices to operate independently and scale based on workload. Additionally, message queues aid in load balancing by evenly distributing messages across multiple instances, optimizing resource utilization. Used in event sourcing, they store events for system history, supporting dynamic service discovery. With transactional features, message queues ensure microservices-based transactions' integrity, forming a critical component for resilient microservices.
As message queue systems are so well-suited to microservice architectures it is very common to see them used in conjunction with containerization and container orchestration technologies, such as Red Hat OpenShift, Kubernetes, VMware Tanzu and Docker amongst others.
A message queue is a fundamental component that enables communication between different components or systems by storing and managing messages in a sequential manner. It provides a basic mechanism for asynchronous communication, allowing one component to send a message to a queue, and another component to retrieve and process the message. The Linux message queues of POSIX and System V IPC are truer to this conventional definition than the message queue systems commonly seen in modern microservice architectures.
On the other hand, a message broker is a more advanced entity that goes beyond basic queuing functionality. It acts as an intermediary in message-based communication between different components or systems. In addition to queuing, a message broker provides additional features such as message routing, translation, and support for various messaging patterns like publish-subscribe and point-to-point.
In essence, while a message queue is primarily focused on the storage and retrieval of messages in a sequential manner, a message broker extends this functionality by providing additional features for managing and facilitating communication between different components or systems in a distributed environment.
Indeed, many of the popular message queue systems define themselves as brokers. RabbitMQ describes itself as “RabbitMQ is a messaging broker - an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received”. In practice common terminology often uses “Message Queue” to refer to systems that are a “Message Broker”.
Many “Message Queue Systems” go further and describe themselves and their broad technologies in terms of services and platform, for example Apache Kafka describes itself as “an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications”.
To proactively monitor message queue health and performance data, tools such as eG Enterprise have key capabilities such as:
Learn more about eG Enterprise’s support for messaging queue monitoring, here: Message queue monitoring | eG Innovations.
The metrics to monitor depend on the specific characteristics and requirements of your message queue system. You should always pick a tool that explicitly supports your message queue technology of choice. However, here is a general list of metrics that a comprehensive monitoring tool for message queues would usually capture and / or alert on:
